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EXECUTIVE  SUMMARY 


Part  of  our  College  mission  is  distribution  of 
the  students'  problem  solving  products  to 
DOD  sponsors  and  other  interested  agencies 
to  enhance  insight  into  contemporary, 
defense  related  issues.  While  the  College  has 
accepted  this  product  as  meeting  academic 
requirements  for  graduation,  the  views  and 
opinions  expressed  or  implied  are  solely 
those  of  the  author  and  should  not  be 
construed  as  carrying  official  sanction. 


insights  into  tomorrow 


REPORT  NUMBER  88-2440 
AUTHOR(S)  MAJOR  WILLIAM  T.  SMITH,  USAF 

TITLE  88TB i  THE  IMPORTANCE  OF  EARLY  TEST  AND  EVALUATION 
ORGANIZATIONAL  PARTICIPATION 

I.  Purpose :  To  substantiate  the  need  to  involve  all 
participating  organizations  early  in  the  Test  and  Evaluation 
<T&E)  process. 

II.  Problem:  It  ie  the  belief  of  Congress  and  users  that  more 
and  more  Department  of  Defense  <DoD>  weapon  systems  are  making 
it  into  production  without  adequate  testing.  As  a  result,  the 
early  involvement  of  Operational  Test  and  Evaluation 
participants  in  the  Test  and  Evaluation  <T&E>  process  has  been 
emphasized.  While  this  is  helpful,  it  is  not  sufficient.  The 
early  involvement  of  all  participating  organizations  in  the  T&E 
process  is  important  to  the  success  of  the  weapon  system 
development  program  and  needs  to  be  emphasized. 

III.  Discussion:  In  the  T&E  process,  it  is  not  enough  Just  to 
understand  the  technology  and  test  requirements  of  the  weapon 
system  itself.  It  is  also  necessary  to  understand  the  technology 
and  test  requirements  of  the  ground  test  support  facility  which 


CONTINUED- 


will  be  used  to  evaluate  the  weapon  system.  It  is  necessary  to 
ensure  the  test  facility  will  be  ready  when  the  weapon  system  is 
ready  to  test.  The  identification  of  these  technological  and 
test  needs  require  early  organizational  participation.  This  is 
particularly  true  when  the  T&E  program  involves  the  development 
and  acquisition  of  a  space  system.  The  space  environment 
requires  that  the  system  works  the  first  time  it  is  used  in 
space.  The  substantial  cost  of  space  hardware  and  the  high 
degree  of  technology  involved  prohibit  the  use  of  the  customary 
incremental  buildup  approach  to  testing.  The  space  hardware 
must  be  carefully  and  completely  tested  on  the  ground  to  protect 
the  large  investment  in  that  item.  This  study  focuses  on  the 
guidance  provided  by  DoD  and  the  Air  Force  regarding  T&E.  It 
also  presents  the  Space  Surveillance  and  Tracking  System  <SSTS> 
development  effort  to  illustrate  the  importance  of  involving  all 
participating  organizations  early  in  the  program. 

IV.  Cone 1 us i ona :  Acquisition,  together  with  the  T&E  process, 

is  well  structured.  The  guidance  provided  by  DoD  and  Air  Force 
is  clear  and  comprehensive.  It  stresses  the  need  for  early  T&E 
and  the  importance  of  the  Test  Planning  Working  Group  (TPWG)  and 
Test  and  Evaluation  Master  Plan  <TEMP) .  In  addition,  since 
adequate  test  facilities  can  take  as  long  to  develop  and  acquire 
as  the  system  itself,  test  facility  requirements  must  be 
identified  early  in  the  development  program.  The  early 
involvement  by  T&E  experts  identified  the  need  for  infrared  (IR) 
scene  generation  to  properly  test  the  focal  plane  array  in  the 
SSTS  development  program.  As  a  result,  the  technology  for  an 
scene  generator  will  have  sufficient  time  to  develop  so  that  an 
adequate  test  facility  will  be  available  when  the  system  is 
ready  to  test. 

V.  Recommendations:  Increased  emphasis  should  be  given  to 

combining  Development  Test  and  Evaluation  (DT&E>  and  Operational 
Test  and  Evaluation  <OT&E>  as  early  as  passible  and  wherever 
possible,  without  compromising  the  individual  test  objectives  in 
the  T&E  process.  In  addition,  Program  Office  Managers  should 
actively  consult  with  test  facility  experts.  This  may  require  a 
change  in  attitude  of  both  parties  regarding  program  obligation 
and  test  facility  proliferation.  Finally,  other  programs  should 
follow  the  SSTS  program  example  in  their  emphasis  on  TPWG,  TEMP 
and  early  organizational  participation. 


Chapter  One 


I HTRODUCT I OS 

Weapon  system  testing  stresses  quantity  -  not  quality; 
often  conducted  under  nonrealistic  conditions;  are 
often  "success  oriented";  are  not  really  independent; 
problems  have  not  only  been  around  for  a  long  time  but 
are  getting  worse.  <18:9) 

Each  year  the  number  of  success-oriented  unrealistic 
tests  increase. . . .  Host  unfortunately. . .our  combat 
forces  are  not  getting  the  systems  that  they  need  to 
carry  out  their  mission.  Mr.  Chairman,  if  there  was 
only  one  system  in  the  Department  of  Defense  that  was 
not  properly  tested  it  would  be  one  too  many.  <18:38) 

The  first  quote  is  a  congressional  staff  opinion  while  the 
second  is  the  view  of  Lt.  Gen.  James  Hollingsworth,  US  Army 
<retired).  Both  views  were  expressed  at  a  1986  Armed  Services 
Subcommittee  hearing  on  the  Department  of  Defense  Test 
Procedures  where  the  central  issue  was  the  adequacy  of  test 
procedures  in  the  acquisition  process.  It  was  the  belief  of 
Congress  and  the  users  of  these  weapon  systems  that  more  and 
more  systems  were  making  it  into  production  without  adequate 
testing.  <18:1)  These  views  have  resulted  in  an  increased 
emphasis  an  testing  by  the  Department  of  Defense,  especially  the 
early  involvement  of  Operational  Test  and  Evaluation  <QT&E) 
participants.  <17:247)  However,  based  on  over  ten  years 
experience  in  testing,  this  is  not  enough.  It  is  this  author’s 
opinion  that  the  early  involvement  of  all  participating 
organizations  in  the  Test  and  Evaluation  <T&E)  process  is 
important  to  the  success  of  the  weapon  system  development 
program.  These  organizations  should  include,  but  not  be  limited 
to,  those  involved  in  technology  development,  development 
testing  as  well  as  operational  testing.  In  the  T&E  process,  it 
is  not  enough  just  to  understand  the  technology  and  test 
requirements  of  the  weapon  system  itself.  It  is  also  necessary 
to  understand  the  technology  and  test  requirements  of  the  ground 
test  support  facility  which  will  be  used  to  evaluate  the  weapon 
system.  It  is  necessary  to  ensure  the  test  facility  will  be 
ready  when  the  weapon  system  is  ready  to  test.  The 
identification  of  these  technological  and  test  needs  require 
early  organizational  participation.  This  is  particularly  true 


when  the  T&E  program  involves  the  development  and  acquisition  of 
a  space  system.  The  purpose  of  this  study  is  to  substantiate 
the  need  to  involve  all  participating  organizations  early  in  the 
T&E  process. 

Test  and  evaluation  programs  follow  guidelines  defined  in 
DoD  Directive  5000.3  which  states  that  ’’Test  and  evaluation 
shall  begin  as  early  as  possible  in  the  system  acquisition 
process....”  <12:2)  While  T&E  does  begin  early,  the  complete 
test  program  objectives  may  not  be  accomplished  effectively; 
leading  to  insufficient  QT&E  results  which  could  cause  the 
program  to  be  canceled.  One  example  of  such  a  program  was  the 
Army’s  Division  Air  Defense  <DIVAD>  system  which  was  canceled  in 
1985  due  to  problems  based  on  OT&E  results.  (1:13) 

Department  of  Defense  test  programs  are  designated  as 
Development  Test  and  Evaluation  <DT&E>  and  Operational  Test  and 
Evaluation  (OT&E).  The  primary  objective  of  DT&E  is  to 
demonstrate  that  the  system  engineering  design  and  development 
is  complete.  Air  Force  DT&E  is  usually  managed  by  Air  Force 
Systems  Command  (AFSC) .  In  contrast,  the  OT&E  objective  is  to 
evaluate  the  system’s  operational  effectiveness,  reliability  and 
suitability.  (8:3)  These  tests  are  usually  managed  by  the  Air 
Force  Operational  Test  &  Evaluation  Center  CAFOTEC) .  Both 
organizations  have  different  missions  and  compete  for  limited 
DoD  resources. .  The  early  involvement  of  all  participating 
organizations,  particularly  those  associated  with  OT&E,  may  be 
limited  since  OT&E  traditionally  follows  DT&E.  Combining  DT&E 
and  OT&E  is  permitted  by  DoD  directives  but  is  not  encouraged. 
Therefore,  by  the  time  DT&E  is  accomplished,  it  is  possible  that 
only  limited  resources  are  available  or  test  requirements  are 
not  identified  in  sufficient  time  to  allow  adequate  OT&E 
testing.  For  special  acquisition  programs,  such  as  space 
systems,  combining  OT&E  with  DT&E  must  be  emphasized  so  that 
DT&E  can  contribute  to  the  accomplishment  of  the  OT&E 
objectives.  <5: 15) 

The  space  environment  requires  that  the  system  works  the 
first  time  it  is  used  in  space.  The  substantial  cost  of  space 
hardware  and  the  high  degree  of  technology  involved  prohibit  the 
use  of  the  customary  incremental  buildup  approach  to  testing. 
The  space  hardware  must  be  carefully  and  completely  tested  on 
the  ground  to  protect  the  large  investment  in  that  item.  (1:6) 
The  interdependence  of  technology  and  test  efforts  in  a  space 
program  must  be  identified  and  maintained.  The 
interrelationships  in  program  development  were  highlighted  by 
Lt.  Gen.  James  A.  Abrahamson,  Director,  Strategic  Defense 
Initiative  Organization,  in  a  Senate  hearing.  He  stated: 

Development  of  a  space  based  sensor  is  dependent,  in 

part,  on  an  integrated  sensor,  which  is  in  turn 

dependent  on  focal  plane  technology,  which  is  built  on 

technology  supporting  new  development  of  arrays, 


submodules,  and  modules.  Multiply  this  by  nearly  a 
hundred  fold  and  one  begins  to  get  an  idea  of  the 
complex  interrelationships  that  intertwine  throughout 
the  program.  <21:80) 

In  addition,  we  have,  to  date,  demonstrated  only  a  limited  on- 
orbit  repair  capability:  therefore,  these  systems  must  also  be 
highly  reliable  once  they  are  deployed.  This  reliability  must 
be  verified  before  deployment.  T&E  takes  on  even  more  dramatic 
importance  because  of  the  criticality  of  reliability.  One  aspect 
of  ground  testing  space  systems  involves  simulation  testing. 

Surveillance  systems  will  rely  on  realistic  simulation 
tests  to  verify  attainment  of  technical  performance 
specifications  and  effectiveness  before  they  are  deployed. 
(5: 15)  Lt.  Gen.  Abrahamson  painted  out  in  the  1986  hearings 
before  the  House  of  Representatives  Subcommittee  on 
Appropriations  for  Research,  Development,  Test  and  Evaluation: 

It  is  hard  to  overestimate  the  importance  of  the 
generation  of  realistic  threat  models,  the  estimation 
of  the  vulnerability  of  targets  to  the  numerous  kill 
mechanism  options  being  exploited,  and  the  development 
of  strategies,  tactics  and  technology  to  ensure  system 
survivability  to  mission  completion.  These  analyses 
and  .  estimates  will  provide  the  boundaries  for 
measuring  success.  (14:570) 

Mr.  Charles  K.  Watt,  Acting  Director,  Defense  Test  and 
Evaluation,  stated  during  a  House  Armed  Services  hearing  that 
the  capabilities  of  supporting  ground  test  facilities  are  vital 
to  the  development  of  all  weapon  systems.  <16:308)  The 
projected  size  of  satellites  and  the  complexity  of  the  mission 
and  the  simulation  environment  in  which  the  satellite  must  be 
tested  may  preclude  a  full  up  operational  test  of  the  complete 
system  until  future  test  capabilities  are  developed.  <5: 15) 
These  test  capabilities  may  require  significant  technological 
development  lead  time.  The  technology  and  test  requirements  must 
be  identified  and  studied  early  in  the  acquisition  process. 
Adequate  funding  for  testing  programs  is  very  important.  The 
competition  for  funds  is  always  keen,  and  early  involvement  by 
as  many  participating  organizations  as  possible  could  increase 
advocacy  for  the  funding  of  test  programs. 

It  is  this  author's  position  that  space  systems  are 
uniquely  suited  to  illustrate  the  importance  of  involving  all 
participating  organizations  early  in  the  development  program. 
In  this  study,  one  space  system  in  particular,  the  Space 
Surveillance  and  Tracking  System  <SSTS) ,  will  be  highlighted  to 
illustrate  that  early  organizational  involvement  is  important  in 
the  development  process. 


The  SSTS  Is  a  very  complex  satellite  surveillance  system 
that  will  be  used  In  our  Strategic  Defense  Initiative  (SDI) 
program.  It's  primary  mission  Is  to  provide  midcourse 
surveillance — acquire,  discriminate,  track  and  perform  kill 
assessment — of  post  boost  vehicles  (PBV) ,  reentry  vehicles  <RV> , 
decays  and  interceptors.  <3: 3-1; 23: — )  One  of  the  nest  critical 
technologies  in  the  system  is  the  development  of  the  focal  plane 
arrays  <FPA) .  (6:4)  The  focal  plane  array  consists  of  the 
sensors  on  the  satellite  that  will  be  used  for  detection  and 
tracking.  It  is  expected  to  track  tens  of  thousands  of  objects 
during  the  midcourse  phase  of  ballistic  missile  flight.  (16:356) 
In  addition,  because  of  the  need  far  greater  precision  tracking, 
the  sensors  will  require  higher  sensitivities  and  therefore  a 
very  large  number  of  these  detector  arrays  will  be  needed. 
(16:214)  It  is  planned  in  the  early  1990s  to  perform  space-based 
surveillance  experiments  to  demonstrate  survivable  means  of 
detecting  and  tracking  boosters  at  very  high  altitudes  in  space 
and  to  determine  our  ability  to  track  large  numbers  of  objects. 
(16:356)  The  ground  testing  of  the  operational  effectiveness  of 
these  focal  plane  arrays  will  range  from  static  radiometric 
calibration  to  performance  testing  against  dynamic  scenes. 
(5: 19)  This  requirement  will  in  turn  require  adequate  test 
capabilities.  Technological  experts  at  Arnold  Engineering 
Development  Center  stated:  "The  capability  to  generate  high 
fidelity  scenes  is  essential  for  all  levels  of  sensor 
test ing. ...  Thus  one  of  the  most  stressing  and  urgent  of  the 
technology  areas  ...is  the  IR  [infrared]  scene  generation.” 
(5:41)  Realistic  representations  of  the  threats  and 
environments  in  which  the  space  system  will  operate  must  be 
tested  before  the  system  is  deployed.  This  will  require  early 
operational  testing  in  the  acquisition  process.  (17:241) 
Further  discussion  of  the  SSTS  is  presented  in  chapter  three. 

As  stated  earlier,  an  understanding  of  the  technology  and 
test  requirements  for  both  the  weapon  system  and  its  ground  test 
facility  should  provide  the  T&E  community  an  increased  awareness 
of  the  importance  of  involving  all  participating  organizations 
early  in  the  development  program.  This  understanding  of 
technology  and  test  needs  requiring  early  organizational 
involvement  will  be  accomplished  by  first  briefly  reviewing  the 
acquisition  process  and  guidance  provided  by  DoD  and  the  Air 
Force.  It  will  show  that  this  guidance  is  quite  clear  and 
comprehensive  regarding  T&E.  Second,  the  T&E  process,  composed 
of  DT&E  and  OT&E,  will  be  discussed  with  the  emphasis  on 
combining  DT&E  and  OT&E  early  in  the  process,  especially  for 
space  systems.  The  importance  of  the  Test  and  Evaluation  Master 
Plan  (TEMP)  and  the  Test  Planning  Working  Group  (TPVG)  to  the 
overall  T&E  process  will  also  be  presented.  Third,  the 
necessity  for  early  identification  of  adequate  ground  test 
facility  support  will  be  presented.  It  will  be  suggested  that  a 
possible  change  in  attitude  of  the  program  office  and  testing 
organizations  may  be  needed  in  order  to  have  an  active  dialogue 
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between  the  two.  Finally,  using  the  SSTS  as  an  example,  it  will 
be  shown  that  the  early  involvement  by  T&E  experts  has  allowed 
this  critical  and  complex  program  time  to  develop  the  focal 
plane  array  and  scene  generator  concurrently;  thus,  the  test 
facility  will  be  ready  when  the  system  is  ready  to  test. 

Several  assumptions  and  limitations  have  been  made  to  limit 
the  scope  of  this  study.  First,  the  study  will  be  limited  to 
the  technology  aspects  of  the  focal  plane  array  and  scene 
generation  as  they  apply  to  the  SSTS.  The  specific  SSTS 
technology  costs  will  not  be  included.  Second,  the  study  will 
be  restricted  to  the  unclassified  aspects  of  SSTS.  Finally  it 
is  assumed  that  the  technology  and  test  programs  are  allowed  to 
progress  in  a  peacetime  environment  (i.e.  no  "crash”  system 
development  programs) . 

The  study  consists  of  three  additional  chapters.  Chapter 
two  deals  with  the  general  acquisition  process  and  specific  DoD 
and  Air  Force  policies  for  test  and  evaluation.  The  objective  of 
T&E  and  its  management  will  be  addressed  along  with  the 
importance  of  the  TEMP  and  TPVG.  Test  facilities  will  also  be 
discussed.  Chapter  three  will  highlight  the  SSTS  and  discuss 
the  technological  and  test  requirements  of  the  focal  plane  array 
and  the  scene  generator.  In  addition,  the  early  participation 
by  T&E  experts  will  be  presented.  Chapter  four  will  conclude 
the  study  with  the  author's  conclusions  and  recommendations. 
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Chapter  Two 


TEST  &  B VALUATION  (TAB) 

I  j  XHB 

ACQUISITION  PROCESS 


ACQUISITION  OF  MAJOR  DEFENSE  SYSTEMS 

Before  addressing  the  Space  Surveillance  and  Tracking 
System  <SSTS>  specifically,  one  must  understand  how  the 
Department  of  Defense  and  the  Air  Force  in  particular  expect 
systems  to  be  developed  and  acquired.  This  chapter  will  deal 
with  the  general  acquisition  process  and  specific  DoD  and  Air 
Force  policies  regarding  test  and  evaluation.  The  Test  and 
Evaluation  (T&E)  process  will  be  discussed  along  with  the 
importance  of  the  Test  and  Evaluation  Master  Plan  (TEMP)  and  the 
Test  Planning  Working  Group  (TPWG) .  A  discussion  on  test 
facilities  with  respect  to  the  T&E  process  will  also  be 
presented. 

Regarding  major  system  acquisitions,  DoD  Directive  5000. 1 
states: 

It  is  the  policy  of  the  Department  of  Defense  to 
ensure  that  DoD  acquisition  of  major  defense  systems 
is  carried  out  efficiently  and  effectively  to  achieve 
the  operational  objectives  of  the  U.S.  Armed  Forces  in 
support  of  national  policies  and  objectives. . . .  (10: 1) 

To  ensure  that  major  systems  are  acquired  in  the  most  efficient 
and  effective  manner,  an  acquisition  strategy  is  developed  at 
the  inception  of  each  major  program.  This  strategy  is  used  to 
establish  the  objectives,  resources,  management  assumptions,  and 
program  structure  (i.e.  development  phases,  decision  milestones, 
test  and  evaluation  periods,  planned  concurrence,  and  production 
releases).  (10:2) 

The  program  structure  is  divided  into  distinct  phases  of 
the  acquisition  process.  These  phases  include:  concept 
exploration,  demonstration  and  validation,  full-scale 
development,  and  production  and  deployment.  According  to  DoD 
Directive  5000. 1,  each  phase  is  "tailored  to  fit  each  program  to 
minimize  acquisition  time  and  cost,  consistent  with  the  need  and 


degree  of  technical  risk  involved.”  (10:4)  A  general  program 
structure — acquisition  process — is  depicted  in  Figure  1.  along 
with  T&E  involvement. 
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Figure  1.  Acquisition  Process  and  T&E  Involvement  (17:249) 

Prior  to  the  concept  and  exploration  phase,  a  mission  need 
determination  is  accomplished  in  the  the  Planning,  Programming, 
and  Budgeting  System  (PPBS)  process.  The  need  is  based  on  the 
Justification  for  Major  System  Mew  Starts  (JMSNS)  submitted  with 
the  Program  Objective  Memorandum  (POM) .  The  Secretary  of 
Defense  provides  guidance  for  the  program  in  the  Program 
Decision  Memorandum  (PDM) .  The  PDM  permits  a  major  system  new 
start  and  allows  the  Military  Services  to  enter  the  concept 
exploration  phase  of  the  acquisition  process.  <10:4; 11:4) 

In  order  to  enter  subsequent  phases,  the  program  must  pass 
specific  milestone  decision  points.  "The  first  Secretary  of 
Defense  major  milestone  decision  is  the  concept  selection  and 
entry  into  the  demonstration  and  validation  phase."  (10:4)  The 
Milestone  I  decision  point  is  used  to  summarize  the  results  of 
the  concept  exploration  phase  and  includes  the  identification  of 
the  concepts  that  will  be  further  pursued  in  the  next  phase. 
The  Test  and  Evaluation  Master  Plan  (TEMP)  is  required  at 
Milestone  I.  TEMPs  will  be  described  in  more  detail  later  in 
this  chapter.  "The  second  Secretary  of  Defense  major  decision 
is  program  go-ahead  and  approval  to  proceed  with  full-scale 
development."  (10:5)  The  purpose  of  the  Milestone  II  decision 
is  to  review  current  system  performance,  revalidate  the  threat 


for  which  the  system  is  required,  and  review  the  overall  cost, 
schedule,  and  testing  program.  These  reviews  are  performed  in 
an  effort  to  reduce  the  risk  and  uncertainty  in  a  program  before 
major  resources  are  committed  to  full-scale  development. 
<10: 5; 11: 4-5)  "The  production  decision  at  Milestone  III  is 
delegated  to  the  DoD  Components,  provided  the  thresholds 
established  at  Milestone  II  are  met."  <10: 5)  It  is  clear  that  a 
well  structured  process  for  system  acquisition  has  been 
established  by  the  Department  of  Defense. 


T&E  AMD  TEMP 

The  T&E  effort  takes  place  throughout  the  acquisition 
process.  As  DoD  Directive  5000.3  states: 

The  primary  purpose  of  all  T&E  is  to  make  a  direct 
contribution  to  the  timely  development,  production, 
and  fielding  of  systems  that  meet  the  user’s 
requirements  and  are  operationally  effective  and 
suitable .  <12:2) 

Test  and  evaluation  starts  at  the  earliest  possible  time  in  the 
acquisition  process  to  reduce  risks  and  to  estimate  the  system’s 
capability  to  meet  all  technical  and  operational  requirements. 
The  testing  should  provide  the  quantitative  data,  while 
minimizing  the  need  for  subjective  evaluations  of  the  system’s 
performance.  <12:4-5)  Development  Test  and  Evaluation  <DT&E> 
and  Operational  Test  and  Evaluation  <0T&E>  make  up  the  T&E 
effort. 

DT&E  is  conducted  throughout  the  acquisition  process.  Its 
primary  purpose  is  to  ensure  that  the  engineering  design  and 
development  of  a  system  is  complete  and  that  it  meets  or  exceeds 
the  technical  performance  specifications.  <12:4)  "The  T&E 
encompasses  the  use  of  models,  simulations,  and  testbeds,  as 
well  as  prototypes  of  full-scale  engineering  development  models 
of  the  system."  <12:4)  OT&E  Is  also  conducted  throughout  the 
acquisition  process  with  the  primary  purpose  of  "ensuring  that 
only  operationally  effective  and  suitable  systems  are  delivered 
to  the  operating  forces."  <12:5)  These  tests  are  performed 
under  realistic  conditions  that  simulate, as  closely  as  possible, 
the  actual  threat  environment.  Additionally,  the  components 
that  are  tested  must  be  representative  of  the  product  to  be 
delivered  by  the  contractor.  <8: 7)  OT&E  is  traditionally 
separate  and  usually  follows  DT&E.  However,  combining  DT&E  and 
OT&E  is  permitted.  DoD  Directive  5000.3  states:  "Combined  DT&E 
and  OT&E... may  be  used  when  cost  and  time  benefits  are 
significant  and  are  clearly  identified,  provided  that  test 
objectives  are  not  compromised."  <12:8)  The  needs  of  both  the 
developing  agency  performing  the  DT&E,  and  those  of  the 
operational  test  agency  performing  the  OT&E,  must  be  satisfied. 
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DoD  Directive  5000.3  also  points  out  that  T&E  of  special 
programs,  such  as  space,  where  the  procurement  of  a  few  items 
over  an  extended  period  represent  a  substantial  resource 
Investment,  the  operational  test  agency  ’’shall  monitor  and 
participate  in  relevant  laboratory  and  controlled  testing,  and 
use  these  test  results,  as  appropriate,  to  provide  an  assessment 
of  system  effectiveness  and  suitability.”  <12: 8)  ”If  separate 
testing  would  cause  significant  delays  or  increases  in  system 
acquisition  cost  or  in  resources,  development  and  operational 
testing  will  be  combined.”  <8: 7)  The  guidance  again  appears 
quite  clear  that  consideration  of  combining  DT&E  and  OT&E  must 
be  accomplished  early  in  the  T&E  process,  especially  regarding 
space  systems. 

One  of  the  most  important  documents  in  the  T&E  process  is 
the  Test  and  Evaluation  Master  Plan  (TEMP).  The  TEMP  is  used  to 
specify  development  and  operational  events.  A  TEMP  is  required 
on  all  Air  Force  programs  directed  by  a  Program  Management 
Directive  <PMD) .  (8:5)  Air  Force  Regulation  80-14  states:  "The 
TEMP  will  integrate  critical  issues,  test  objectives,  evaluation 
criteria,  system  characteristics,  responsibilities,  resources, 
and  schedules  for  T&E.”  <8: 5)  Approved  TEMPs  are  required 
before  the  start  of  any  system  testing  and  are  updated  at  least 
annually.  <8: 6)  The  TEMP  is  such  an  important  document  in  T&E 
that  the  Under  Secretary  of  Defense,  Research  and  Engineering 
(USDR&E) ,  Dr.  James  P.  Vade  Jr.,  emphasized  during  a  1985  House 
Armed  Services  hearing,  the  continuing  need  for  quality 
Improvements  in  the  preparation  of  TEMPs  to  ensure  the 
completion  of  essential  tests  on  T&E  programs.  (16: 153)  He  has 
asked  the  Services  to  continue  to  improve  the  timeliness  and 
quality  of  the  TEMPs  "particularly  in  quantifying  system 
parameters  to  be  verified  by  testing.”  (16:153)  The  early 
identification  of  test  resource  requirements  and  possible 
shortfalls  and  contingency  plans  to  overcome  the  shortfalls  are 
now  a  mandatory  part  of  the  TEMP.  (16: 153)  The  next  year,  Mr. 
John  E.  Krings,  Director  of  Operational  Test  and  Evaluation 
(DOT&E) ,  at  another  House  Armed  Services  hearing,  expressed  his 
views  on  the  importance  of  the  TEMP  when  he  stated  that  DOT&E 
would  use  "its  authority  under  the  law  to  approve  and  disapprove 
TEMPs  and  OT  [Operational  Test]  plans  to  assure  that  OT  concerns 
and  requirements  are  incorporated  early  in  the  life  of  a  new 
program.”  (17:241)  The  TEMP  is  considered  a  living  document, 
its  currentness  is  so  essential  that  the  Secretary  of  Defense 
will  direct,  in  the  revised  DoD  Directive  5000.3,  that  "...no 
program  may  proceed  to  a  Defense  System  Acquisition  Review 
Council  (DSARC)  or  Service  SARC  review  unless  it  has  an  up-to- 
date  TEMP  approved  by  DOT&E  and  USDR&E.”  (17:243-244)  In 
chapter  three  we  will  see  that  for  the  SSTS  the  technology  and 
test  needs  were  well  documented  in  the  TEMP. 


MANAGEMENT  OF  DT&E  AND  OT&E 

Many  organizations  can  become  involved  in  the  T&E  process; 
however,  who  manages  the  DT&E  and  OT&E  and  how  do  other 
organizations  participate?  Air  Force  Regulation  80-14  states: 
*'The  implementing  command  is  responsible  for  DT&E  management." 
<8: 6)  The  Implementing  command  for  Air  Force  acquisitions  is 
the  Air  Force  Systems  Command  (AFSC) .  Among  the  most  important 
responsibilities  of  the  Implementing  command  is  that  they  plan, 
manage,  and  conduct  the  DT&E;  they  integrate  the  OT&E 
requirements  into  the  acquisition  process;  they  establish  and 
chair  the  Test  Planning  Working  Group  <TPWG>;  and  they  prepare 
and  coordinate  the  TEMP.  <8: 14 > 


Management  of  the  Air  Force  OT&E  program 
responsibility  of  the  Air  Force  Operational  Test  and 

Center  (AFOTEC).  (8:6)  However,  HQ  USAF  can  designate 
the  operating  command  as  the  OT&E  command.  <8: 11) 
most  Important  responsibilities  of  the  OT&E  command 
they  accomplish  the  HQ  USAF  directed  OT&E  and  are 
involved  in  DT&E  to  obtain  Information  applicable 
objectives.  In  addition,  they  serve  as  a  member  of 
when  AFOTEC  is  designated  the  OT&E  command.  (8: 11-12) 
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The  TPVG  includes  representatives  from  each  organization 
involved  in  the  test  program.  The  members  could  include:  program 
office,  test  agencies,  various  commands  (operating, 
participating  etc.),  and  contractors.  It  is  formed  and  chaired 
by  the  implement ing  command.  Its  purpose  is  to  provide  a  forum 
to  actively  discuss  T&E-related  subjects.  As  stated  in  Air  Force 
Regulation  80-14:  "It  helps  to  set  test  objectives  and 
evaluation  baselines,  and  defines  organizational 
responsibilities  and  relationships."  (8:5)  It  is  "...formed  in 
time  to  allow  for  early  and  thorough  test  planning  and 
coordinating  with  proper  agencies. ...  assists  in  updating  the 
TEMP  and  ...monitors  test  progress.”  (8:5)  The  TPVG  provides  a 
unique  opportunity  for  all  participating  organizations  to  get 
involved  early  in  the  development  program.  As  we  shall  see  in 
chapter  three,  the  program  office  for  the  SSTS  made  excellent 
use  of  the  TPVG. 


MAJOR  RANGE  AND  TEST  FACILITY  BASE  (MRTFB) 

Test  organizations  providing  the  T&E  support  facilities 
make  up  the  Major  Range  and  Test  Facility  Base  (MRTFB).  DoD 
Directive  3200. 11  states:  "The  MRTFB  is  a  national  asset  which 
shall  be  sized,  operated  and  maintained  primarily  for  DoD  test 
and  evaluation  support  missions....”  (9:1)  The  capabilities  of 
these  ranges  and  facilities  are  based  on  a  combination  of  user 
requirements  and  the  mission  of  the  individual  MRTFBs.  (9:2) 
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Few  test  facilities  must  go  through  a  thorough  process  that 
includes  planning  and  technology,  design,  construction, 
activation  and  operation.  This  process  could  take  as  long  as  a 
major  system  program  acquisition  and,  therefore,  must  be  started 
as  soon  as  the  need  is  Identified.  <24: — )  The  Focal  Plane 
Array  Facility  used  for  testing  the  SSTS,  described  in  chapter 
three,  is  estimated  to  take  six  years  to  develop  at  an 
approximate  cost  of  65  million  dollars.  <4:B-157>  Acquisition 
of  a  near  term  operational  SSTS  is  expected  to  take 
approximately  eight  years.  <3:4-1)  The  Honorable  Dr.  Thomas  E. 
Cooper,  Assistant  Secretary  of  the  Air  Force  Research, 
Development  and  Logistics,  recognized  that  in  order  to  support 
the  development  and  production  of  weapon  systems  and  the 
Increased  emphasis  by  Congress  on  testing,  a  substantial 
investment  is  required  to  modernize  and  improve  these 
facilities.  <15: 161)  Unfortunately,  an  organizational  bias  may 
inhibit  cooperation  between  system  program  offices  and  test 
organizations.  A  recent  Air  Command  and  Staff  College  research 
study  has  suggested  that  "organizat ional  relationships  and 
procedural  responsibilities  create  a  built-in  bias  which  thwarts 
efforts  to  obtain  needed  test  range  resources  for  operational 
testing."  <2: — )  According  to  Air  Force  Regulation  80-1, 
military  construction  costs  must  be  included  in  the  system 
development  casts  for  construction  that  is  directly  related  to 
the  system.  <7: 2)  As  a  result,  many  system  program  offices  are 
very  reluctant  to  involve  the  test  range  community  for  fear  that 
this  activity  may  carry  with  it  the  obligation  to  fund  proposed 
test  facilities.  <2: 3)  This  reluctance  can  contribute  to  a 
situation  where  a  program  office  fails  to  take  full  advantage  of 
available  experts  in  the  field  of  testing  and  test  techniques. 
A  change  in  attitude  for  both  the  program  office  and  test 
organization  is  required.  The  program  office  must  remember  that 
the  involvement  of  test  expertise  very  early  in  the  development 
program  does  not  carry  with  it  any  obligation  to  advocate  for  or 
fund  test  facility  proposals  unless  it  is  mutually  agreed  that 
the  program  requires  such  a  facility.  The  test  organization 
must  remember  that  every  new  system  does  not  require  a  new  test 
facility. 


SUMMARY 

The  acquisition  process  is  well  structured  and  is  tailored 
for  each  program  to  minimize  acquisition  time  and  cost  while 
reducing  the  technical  and  operational  risks.  The  goal  is  to 
deliver  a  system  to  the  user  that  will  enable  him  to  accomplish 
his  mission.  The  guidance  set  forth  by  the  DoD  and  Air  Force  is 
quite  clear  and  comprehensive  for  the  acquisition  process  as 
well  as  the  T&E  process.  Test  and  evaluation  must  begin  at  the 
earliest  possible  time  in  the  acquisition  process.  T&E  is  made 
up  of  DT&E  and  OT&E,  each  having  their  own  objectives.  For 
space  systems,  combining  DT&E  and  OT&E  must  be  considered  early 
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in  the  TAB  process.  TBMPs  and  TPVGs  are  very  Important  in  the 
TAE  process.  Both  need  to  be  continually  emphasized  and 
utilized.  Vlth  respect  to  test  facilities,  adequate  test 
facility  support  can  take  as  long  to  develop  and  acquire  as  the 
system  itself;  therefore,  the  test  facility  requirements  must  be 
identified  early.  Finally,  program  offices  and  test 
organizations  need  to  adjust  their  attitude  toward  program 
obligation  and  proliferation  of  new  test  facilities  so  that 
there  can  be  an  active  exchange  of  ideas  and  information  between 
both  organizations. 
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Chapter  Three 


SPACB  SURVEILLANCE  A  HD  TRACK  I HG 
SYSTBM  (SSTS) 


GENERAL 

In  chapter  two  we  reviewed  the  guidance  provided  by  the 
Department  of  Defense  and  the  Air  Force  regarding  testing  and 
the  acquisition  process.  In  this  chapter  we  will  address  one 
space  system  in  particular,  the  Space  Surveillance  and  Tracking 
System  (SSTS) .  The  technology  and  test  requirements  of  the 
focal  plane  array  and  scene  generator  will  be  discussed.  In 
addition,  the  early  participation  by  T&E  experts  will  be 
presented. 

General  Abrahamson,  Director  of  the  Strategic  Defense 
Initiative  Organization,  in  a  Senate  hearing  on  the  Strategic 
Defense  Initiative  <SDI>  program  made  this  statement  about  the 
SSTS  supporting  the  defensive  weapons  in  SDI: 

One  of  the  key  sensors  is  a  space  surveillance  and 
tracking  system  I SSTS]  located  in  medium-earth  orbit 
that  is  capable  of  doing  surveillance  against  all 
objects  that  come  into  space  as  well  as  doing 
surveillance  of  rockets  against  the  harder  Earth, 
[boldface  added]  <22:107) 

The  R&D  program  for  this  system  is  expected  to  cost 
approximately  1.3  billion  dollars  between  1985  and  1989.  <19:15) 
Since  the  SSTS  is  one  of  the  key  sensors  in  the  SDI  program  and 
represents  a  substantial  investment,  it  is  imperative  that  the 
system  operate  effectively.  Successful  T&E  becomes  increasingly 
more  important.  Early  organizational  participation  in  the 
development  program  can  contribute  to  this  success.  The  SSTS  is 
a  very  complex  satellite  surveillance  system  whose  primary 
mission  is  to  provide  midcourse  surveillance  for  SDI.  <3:3-1) 
Before  SDI,  this  system  was  known  as  the  Space-Based 
Surveillance  System  which  was  intended  to  provide  support  to  the 
antisatellite  <ASAT)  programs  in  the  space  defense  role. 
<20: 109)  When  SDI  was  initiated  in  1983,  the  midcourse 
surveillance  role  was  added  and  became  its  primary  role.  To 
accomplish  this  role,  HQ  Space  Division  stated  in  the  "SSTS 
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System  Test  Requirements"  briefing  that  the  SSTS  is  expected  to 
"acquire  and  track  PBVs  [post  boost  vehicles],  RVs  [reentry 
vehicles] ,  decoys  and  interceptors;  discriminate  RVs  from 
decoys;  designate  targets  and  update  interceptors;  perform  kill 
assessments;  handover  information  to  BM  [battle  management]  and 
terminal  defense;  and  survive  and  endure."  (23: — >  In  the  space 
defense  role,  according  to  Dr.  Cooper,  the  SSTS  was  to  "provide 
full  earth  orbit  coverage,  reduce  dependence  on  overseas  basing 
of  sensors,  and  provide  timely,  operationally  responsive 
coverage  of  objects  and  events  in  space."  (16:1075)  The  initial 
operational  time  periods  were  divided  into  the  near  (1994-2000), 
mid  (2000-2005),  and  far  (2005+)  term  with  corresponding 
increases  in  the  operational  performance  of  the  SSTS  as  the 
system  evolves.  (3:4-1) 

FOCAL  PLANE  ARRAYS  (FPA) 

One  of  the  most  critical  technologies  identified  in  the 
TEMP  for  the  SSTS  is  the  development  of  the  focal  plane  arrays 
(FPA).  (6: 4; 23: — )  The  focal  plane  array  consists  of  the 
sensors  on  the  satellite  that  will  be  used  for  detection  and 
tracking.  Since  the  satellite  is  expected  to  track  tens  of 
thousands  of  objects  during  midcourse,  a  very  large  number  of 
sensors  are  anticipated.  As  stated  in  the  TEMP:  "Test  facilities 
to  perform  integrated  150, 000  element  FPA  and  processor  testing 
will  be  needed."  (6:18)  In  addition  to  the  development  of  a 
large  number  of  these  detectors  in  a  cost  effective  manner, 
other  technology  Issues  are  involved,  such  as,  radiometric 
performance  and  hardening,  producibility/stability,  and  analog 
preprocessing.  (23: — )  Various  approaches  to  deal  with  these 
Issues  have  been  identified  under  a  focal  plane  technology 
program.  (23: — ;13:34)  The  testing  on  the  FPA  has  been 
summarized  in  the  TEMP  and  in  part  states: 

FPA  testing  evolves  from  individual  detector  testing 
to  selected  FPA  modules  and  finally  a  complete 
FPA. . .and  the  total  of  120,000  detectors. . . .These 
tests  are  most  likely  performed  at  the  contractors 
facilities.  The  next  level  of  integration  will  be  a 
complete  FPA  with  optics  in  the  telescope  test. . . .New 
or  modified  facilities,  instrumentation,  and  perhaps 
new  testing  and  measuring  techniques  will  be  applied. 
Testing  of  the  complete  telescope  will  include, 
optical  performance  and  alignment,  thermal,  and 
mechanical  characteristics.  After  completion  of  these 
tests  the  FPA  will  be  installed  to  make  up  the 
complete  telescope  for  radiometric  testing.  ...  To 
perform  the  radiometric  performance  tests  the  facility 
at  AEDC  [Arnold  Engineering  Development  Center]  will 
(be]  enlarged  to  accept  the  one  meter  class  aperture 
telescope. . . .  (6: 16) 


SCENE  GENERATION 


The  TEMP  also  Identified  test  limitations  for  the  FPA  due 
to  limitations  in  the  state-of-the-art  In  Infrared  scene 
generation.  (6:13)  The  TEMP  states:  "The  major  operational 
issues  associated  with  the  SSTS  system  concern  the  ability  to 
detect  and  track  targets  and  then  report  the  appropriate 
messages  to  the  service  weapon  systems  at  the  proper  time." 
<6: 11)  The  SSTS  is  expected  to  track  a  large  number  of  targets 
in  space  under  strategic  defense.  Since  this  operation  can  not 
be  fully  tested  in  space,  some  of  this  OT&E  will  be  ground 
tested.  Mission  scenarios  involving  a  large  number  of  targets 
under  varying  conditions  will  be  tested  in  ground  simulators. 
<6: 11)  An  AEDC  Space  Plan  providing  guidance  in  the  development 
of  T&E  improvements  for  space  systems  stated: 

The  capability  to  generate  high  fidelity  scenes  is 
essential  for  all  levels  of  sensor  testing  from  focal 
plane  to  full  sensor  system.  Thus,  one  of  the  most 
stressing  and  urgent  of  the  technology  areas  which 
must  be  addressed  is  that  of  IR  scene 
generation.  <5:41) 

Scene  generators  must  provide  realistic  radiance  patterns 
to  the  sensors.  These  patterns  must  simulate  the  real  world 
radiance  patterns  "...with  a  fidelity  such  that  the  sensor. .. is 
unable  to  detect  the  difference."  <4:5.1-51)  The  scenes  must  be 
dynamic;  simulating  the  motion  due  to  the  movement  of  the  line 
of  sight,  background  objects  and  the  targets  themselves.  AEDC, 
in  their  "Space  Facility  Conceptual  Design"  study,  stated: 
"Scene  generator  technology  development  must  begin  with  the 
infrared  source  plane.  Mo  demonstrated  capability  was  found  for 
this  critical  component."  <4:5.3-53)  It  was  estimated  by  AEDC 
that  it  would  take  four  years  to  develop  the  appropriate  scene 
generator  at  a  cost  of  approximately  20  million  dollars.  <4:B- 
143)  The  identification  of  the  need  for  scene  generation 
technology  and  its  development  can  be  attributed  to  the  early 
participation  by  experts. 


EARLY  PARTICIPATION  BY  T&E  EXPERTS 

The  Air  Force  Space  Division  is  responsible  for  the 
development  of  SSTS.  The  SSTS  Program  Office  is  responsible  for 
managing  the  DT&E  while  AFOTEC  will  most  likely  manage  OT&E. 
The  Space  Technology  Center  is  responsible  for  the  focal  plane 
development.  <24: — ;6:4>  HQ  Space  Division  invited  all 
agencies,  including  test  facility  experts,  to  the  TPVG  meetings. 
The  T&E  experts  at  the  TPWG  meetings  concluded  that  there  was  no 
existing  capability  to  test  integrated  mosaic  focal  plane 
assemblies  with  complex  IR  scenes  and  realistic  space 
backgrounds.  In  addition,  it  was  determined  that  current 


infrared  test  facilities  were  limited  in  size.  <24: — >  The 
identification  of  these  inadequate  test  capabilities  resulted  in 
early  studies  to  address  this  deficiency.  <4: — ;5:-->  Had  not 
all  the  agencies  participated  in  the  TPVG,  this  deficiency  may 
have  been  overlooked.  The  passible  consequence  of  overlooking 
this  deficiency  in  the  T&E  process  would  have  been  the  risk  of 
placing  a  critical  surveillance  system  asset  in  orbit  without 
adequate  ground  testing.  <24: — ) 

SUMMARY 

The  SSTS  program  has  clearly  followed  the  guidance  set 
forth  in  the  DoD  directives  and  Air  Force  regulations  discussed 
in  chapter  two.  They  made  excellent  use  the  the  TEMP  and  TPVG. 
It  should  be  noted  that  although  no  OT&E  manager  was  designated, 
the  TPVG  and  TEMP  completely  outlined  the  OT&E  effort  and 
recognized  the  limitation  in  infrared  scene  generation.  The 
combining  of  DT&E  and  OT&E  will  be  performed  where  possible. 
The  program  office  was  not  reluctant  to  involve  the  testing 
agencies  early  in  the  development  program  to  help  establish  the 
test  requirements.  Finally,  and  most  importantly,  they  involved 
all  participating  organizations  early  in  the  development 
program. 
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>  CONCLUSIONS 
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This  study  has  briefly  reviewed  the  acquisition  process  and 
the  guidance  provided  by  DoD  Directives  and  Air  Force 
Regulations  regarding  the  T&E  process.  The  importance  of  the 
Test  Planning  Working  Group  (TPVG)  and  Test  and  Evaluation 
Master  Plan  (TEMP)  was  highlighted.  The  adequacy  of  ground  test 
facility  support  was  also  discussed.  Finally,  the  SSTS  was 
presented  as  an  example  of  early  organizational  participation. 
These  areas  were  presented  to  provide  the  T&E  community  an 
understanding  that  technology  and  test  requirements  for  both  the 
weapon  system  and  its  ground  test  support  facility  require  the 
early  involvement  of  all  participating  organizations  in  the 
development  program.  Specific  conclusions  are  as  follows: 


1.  Acquisition,  together  with  the  T&E  process,  is  well 
structured.  The  guidance  provided  by  DoD  and  Air  Force  is  clear 
and  comprehensive.  It  stresses  the  need  for  early  T&E  and  the 
importance  of  the  TPWG  and  TEMP. 


2.  Adequate  test  facilities  can  take  as  long  to  develop 
and  acquire  as  the  system  itself;  therefore,  the  test  facility 
requirements  must  be  identified  as  soon  as  possible  in  the 
development  program. 


3.  Space  systems  are  very  expensive  to  acquire;  they 
involve  high  technology;  and  have  very  complex  missions  to 
perform.  They  are  special  in  the  T&E  community  because  they 
must  work  the  very  first  time  they  are  deployed  in  space.  This 
means  that  most  of  the  space  hardware  must  be  carefully  and 
completely  tested  on  the  ground.  If  the  current  ground  test 
capabilities  are  inadequate  to  properly  test  the  system,  then 
future  test  capabilities  must  be  identified,  studied  and 
developed.  This  requires  the  early  involvement  of  testing 
experts.  In  the  case  of  the  SSTS,  the  early  organizational 
participation  identified  the  need  for  infrared  <IR>  scene 
generation  to  properly  test  the  focal  plane  array.  As  a  result, 
the  technology  for  an  IR  scene  generator  will  have  sufficient 
time  to  develop  so  that  an  adequate  test  facility  will  be 
available  when  the  system  is  ready  to  test. 
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RECOMMEND AT I OHS 


1.  Increased  emphasis  should  be  given  to  combining  DT&E 
and  OT&E  as  early  as  possible  and  wherever  possible,  without 
compromising  the  individual  test  objectives  in  the  T&E  process. 

2.  Program  Office  Managers  should  actively  consult  with 
test  facility  experts.  This  may  require  a  change  in  attitude  of 
both  parties  regarding  program  obligation  and  test  facility 
proliferation. 

3.  Other  programs  should  follow  the  SSTS  program  example 
in  their  emphasis  on  TPVG,  TEMP  and  early  organizational 
participation. 

CLOSING  REMARKS 

The  need  to  adequately  test  every  system  that  is  being 
acquired  cannot  be  overemphasized.  The  time  to  find  out  if  the 
weapon  system  performs  effectively,  reliably,  and  suitably  is 
not  in  the  heat  of  the  battle.  Ve,  in  the  T&E  community,  have 
an  obligation  to  develop  and  provide  to  our  users  the  most 
efficient  and  effective  weapon  system  we  can.  This  means  we 
must  do  everything  possible  to  test  the  weapon  system  as  soon  as 
possible  in  the  most  realistic  environment.  This  means  a  team 
effort  where  all  the  players  are  involved  and  can  contribute  to 
the  program  from  the  start — early  organizational  participation. 
Finally,  we  must  all  remember  the  words  of  General  Hollingsworth 
presented  at  the  beginning  of  this  study,  ” . . . if  there  was  only 
one  system  in  the  Department  of  Defense  that  was  not  properly 
tested  it  would  be  one  too  many.”  (18:38) 
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